Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TASK: Use saucectl with configuration #3851

Merged
merged 23 commits into from
Sep 20, 2024
Merged

TASK: Use saucectl with configuration #3851

merged 23 commits into from
Sep 20, 2024

Conversation

markusguenther
Copy link
Member

@markusguenther markusguenther commented Sep 17, 2024

Overview:

We are using sauce labs as a remote browser service. The versions we were using are on an API version that has been deprecated. So this Pull Request brings the sauce labs usage to the latest versions. For that, we are now using the CLI tool of sauce labs saucectl. To be able to access the remote browsers, saucectl needs on CircleCI a tunnel. So we also need to use the Sauce Connect Proxy 5 CLI

Changes:

  1. Use saucectl CLI Tool:

    • We have transitioned to using the official saucectl CLI for managing and executing Sauce Labs tests.
    • saucectl runs the testcafe tests
    • To enable access to remote browsers on CircleCI, we now utilize the Sauce Connect Proxy 5 CLI tool, which creates the necessary tunnel for Sauce Labs.
  2. Removal of Sauce Labs Test Links in PR Comments:

    • The current approach of adding links to Sauce Labs test results in PR comments was unreliable and has been removed.
    • In the future, we will look into leveraging the new features from the latest tools, such as webhooks, to automatically add test-related information back to the PR.
  3. New Features Enabled:

    • The updated tools introduce several improvements:
      • Enhanced test management: Test cases now appear directly in the Sauce Labs dashboard.
      • Improved test failure analysis: You can now jump directly to failed test videos.
      • Simplified multi-browser testing: It is now easier to add and manage test suites across multiple browsers.

New view of tests:
Screenshot 2024-09-19 at 09 56 01

Old view of tests:
Screenshot 2024-09-19 at 09 55 35

Future Improvements:

With these updates, we have laid the groundwork for:

  • Automating test information reporting in PRs through webhooks.
  • Extending our testing to more browsers and devices with minimal configuration.

The validation of the saucelabs config failed
Resolves: ERR failed to execute run command error="'node_modules' is ignored by sauceignore, but you have npm dependencies defined in your project; please remove 'node_modules' from your sauceignore file"
Resolves: sauce_connect proxying is denied to host "127.0.0.1:8081" localhost proxying is disabled
@markusguenther markusguenther force-pushed the task/use-saucectl branch 3 times, most recently from 7d00005 to de6bce0 Compare September 19, 2024 13:56
@markusguenther markusguenther force-pushed the task/use-saucectl branch 3 times, most recently from 3333094 to 6d0e10e Compare September 19, 2024 16:35
@markusguenther markusguenther force-pushed the task/use-saucectl branch 13 times, most recently from e05b62c to f038db6 Compare September 20, 2024 06:29
@markusguenther
Copy link
Member Author

Added another little improvement - at sauce labs we now are able to filter for the target branch of the PR.
That makes it faster and easier to find the right recording.

Screenshot 2024-09-20 at 08 48 47

@markusguenther
Copy link
Member Author

We had a ticket in the past for more browsers in the acceptance tests. I could not find the issue, but which browsers we should test and is it ok for us that the tests take much longer?

Currently, I am using just Firefox, but we can use everything from the table below.
Screenshot 2024-09-20 at 09 26 11

@markusguenther
Copy link
Member Author

As some tests are flaky and just need a rerun occasionally, the new setup does that.
Here, the Windows suite failed in one test case. The Mac test suite started right after the finish of the Windows test suite, but as these failed, the Windows suite runs again. And in the end, everything was green.

Screenshot 2024-09-20 at 09 50 00

Copy link
Member

@skurfuerst skurfuerst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much <3 <3

@skurfuerst skurfuerst merged commit 278d0dc into 8.4 Sep 20, 2024
9 checks passed
@skurfuerst skurfuerst deleted the task/use-saucectl branch September 20, 2024 09:01
@@ -108,7 +108,7 @@ test:

## Executes integration tests on saucelabs.
test-e2e-saucelabs:
bash Tests/IntegrationTests/e2e.sh "saucelabs:Firefox@latest:Windows 10"
bash Tests/IntegrationTests/e2e-saucelabs.sh
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need another script that is almost e2e.sh? that will it make hard to maintain, especially since in neos 9 the script is partly rewritten. So we def need adjustments for Neos 9 ... and it probably better to have a shared script for local and sauce e2e tests?

markusguenther pushed a commit that referenced this pull request Sep 24, 2024
TASK: Use saucectl with configuration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants